A network switch is an intermediate network node that provides a “switching” function for transferring information among entities of a communications network. Typically, the switch is a computer comprising a collection of ports interconnected by a switch fabric. Each port couples the switch to a network entity over any of various types of media, including gigabit Ethernet or Fibre Channel link connections. The network entity may consist of any node, such as an end or intermediate network node, configured to transmit or receive information over the media. The switching function thus generally comprises receiving information at an input port from a network entity, forwarding that information to at least one other output port via the switch fabric and, thereafter, transmitting the information over at least one medium to another entity of the network.
Where only a relatively few input and output ports are needed to receive and transmit data over the communications network at relatively low bandwidth/speeds, the network switch may comprise a single device or module. However, for an application of the switch requiring relatively large numbers of ports (e.g., 64–256) and relatively high bandwidth per port (e.g., 2–10 gigabits per second), a plurality of different modules (e.g., 8–16) may be needed. The modules constituting such a conventional “large” switch may be grouped into three categories: control modules (CM), line modules (LM) and switch modules (SM). The CM provides control and monitoring functions for the operation and health of the switch using low-bandwidth communication paths to all other modules for initialization and maintenance. The LM provides standard format data connections to attach the switch to the network. The SM provides the actual switching function in a switch fabric, such as a conventional crossbar fabric, to transfer data between all LMs in the switch.
One type of crossbar fabric is a variable-sized packet switch fabric adapted to switch packets received over input connections to output connections of the SM. Another type of crossbar fabric used to switch data packets of varying sizes is a cell switch fabric that operates on packets apportioned into fixed-size cells. The fixed sized cells switched by the crossbar fabric are generally smaller than a typical packet size. Upon receiving a packet, an ingress (source) LM apportions that packet into fixed sized cells and forwards those cells to the SM. The cell switch fabric on the SM “switches” the cells and forwards them to an egress (destination) LM, which then reassembles the cells into the packet and forwards that packet over the network.
A cell switch fabric is often less expensive to implement than a variable sized packet switch and facilitates computation of the input-to-output connections. In addition, the cell switch fabric allows for better differentiation of quality of service (QoS) levels and achieves better utilization than the variable sized packet switch. Moreover, the cell switch fabric ensures “fairness” among the variable sized packets switched by the crossbar fabric. That is because of the varying sizes of packets, use of a fixed-sized cell granularity ensures fairness among all packets received at the switch by enabling, e.g., interleaving of cells from large and small packets during the switching function. Packet segmentation and reassembly costs are easily offset by these benefits, such that a cell switch fabric is commonly used in networking equipment.
Since the cell switch fabric operates on fixed sized units or cells, synchronization of the crossbar switch fabric is critical. The cell switch fabric is configured to receive cells from each of the source LMs and forward those cells to one or more destination LMs in the switch. This is generally performed in connection with a fixed “cell time.” A cell time is defined as the time needed to receive a cell at an input port of the switch fabric, connect that input port to an output port of the switch fabric and switch the cell to the output port for transfer to a destination LM. If more than one source LM attempts to send a cell to a particular destination LM, an arbitration policy is computed by an arbiter to decide which input port should be connected to the output port of the destination LM. This arbitration decision is also performed during the cell time.
To efficiently perform the switching function, all cells from all source LMs destined for switching should be present at the input ports of the crossbar switch fabric at the same time so that they can be delivered to the destination LMs at the same time. Crossbar synchronization involves aligning all of the cells from source LMs at the input ports of the switch fabric at the same time so that switching may occur in “walk step.” The crossbar switch fabric is typically implemented as a multiplexor circuit having a plurality of inputs and a single output. The result of the arbitration decision configures the multiplexor to connect a particular input to the output of the multiplexor. In general, there is a single multiplexor resource per output port of the switch fabric. To efficiently operate the multiplexor, all cells must be present (aligned) at the inputs (input ports) of the multiplexor (crossbar fabric) at the time the arbitration result is applied to that circuit.
Configuration of the LM may be simplified by only having connections to and from the SM, which limits the switch size to the number of connections the SM can support. Every connection to and from the SM is therefore as fast as the interconnect technology allows, in order to maximize the switching bandwidth. For a relatively small switch having relatively low bandwidth of ports, the interconnect technology allows data to be transmitted over electrical wires synchronously. For example, a central clock source provides clock signals to the LMs that enable parallel, synchronous clocking of data between the LMs and SM. The data cells are thus transmitted over a data path comprising a parallel bus that extends between the LMs and the SM.
For this simplified approach, the clock source also sends a synchronization pulse equal to the cell time along with the clock signals. The synchronization pulse is used to align the data cells sent over the parallel bus to the cell switch fabric of the SM. However, as the bandwidth per port and/or number of ports increases, this approach degrades and becomes impractical because, e.g., scaling of the number of ports results in an excessive amount of parallel wires that cannot be manufactured into a practical backplane. The connector structure needed to couple electrical wires between the modules also becomes unwieldy and impractical to install on the backplane. Moreover, the integrity of the clock and data signals transferred over the parallel bus at high bandwidths (i.e., the signal integrity) degrades because of, e.g., interference.
A common solution to this problem is to convert the parallel bus to a data path comprising serial interconnect links that incorporate clock forwarding techniques (with separate clock signal lines) to transmit the data and clock signals between the LMs and the SM. The bandwidth of the parallel wires is generally equal to the bandwidth on the high-speed serial link. For example, if there are 8 parallel input wires, each of which provides 1 megabit per second (Mps) of bandwidth, then the resulting high-speed serial link has a bandwidth of 8 Mps. High-speed communication devices, such as specialized transmitters and receivers, then transfer data between the modules by serializing the parallel data into a bit stream, transferring the bit stream from a transmitter to a receiver, and thereafter accumulating the data back into the original parallel format. Unfortunately, these communication devices introduce significant latency and skew into the data path, often spanning many clock periods. That is, as bandwidth increases (e.g., to greater than 100 megahertz) the serial link approach creates signal integrity issues because of skew that results from, e.g., differing serial link lengths.
One way to increase the bandwidth/speed of data transmitted over the serial link is to send the data over the link without the accompanying (separate) clock signals. Here, the transmitter serializes the parallel data into an encoded bit stream that is decoded at the receiver prior to being converted to parallel format. Conventional clock data recovery (CDR) circuitry on the receiver recovers the data from the encoded bit stream. Broadly stated, the CDR circuitry examines transitions within the serial bit stream to recover embedded clock signals used to sample the encoded bit stream and recover the original input frequency of the transmitted data. Therefore, the original input frequency fi of the data at the transmitter is equal to the recovered output frequency fo of the data at the receiver.
However, there is no fixed phase relationship between the frequencies fi and fo. In fact, there are typically phase differences between the frequencies that could extend beyond a single clock period, resulting in “shifting” of recovered data at the receiver. That is, even though the data recovered by the CDR circuitry is identical to that transmitted over the serial link, the recovered data may be shifted in time to thereby create a phase displacement between the input frequency fi and output frequency fo of the data. The phase differences cause cells to arrive at the switch fabric at different times, thereby obviating the attempt to align data at the crossbar fabric for switching at a defined cell time. This misalignment or phase shift may be due to differences in etch lengths between the LMs and SM, along with process, voltage and temperature (PVT) variations among CDR logic within the various receivers in the switch. Moreover, differences among implementations and vendors of the specialized transmitter and receiver devices, as well as pointer initialization of those devices, may result in such misalignment.
In sum, the crossbar fabric of a network switch is efficiently implemented when the cells from all source LMs are exactly aligned in time for switching purposes. However, typical module communication techniques introduce substantial skew into the cell delivery path of the switch; when multiple paths are involved in a serial process, more than an entire cell time's worth of skew is injected into those paths. The present invention is directed to a technique that efficiently enables alignment of cells at a crossbar switch fabric. The invention is further directed to a technique that addresses the skew issue as well as the PVT variations and implementation/vendor differences associated with high-speed serial link transmissions using CDR logic and specialized transmitters/receivers.